

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Monitoring Stack with Cephadm &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/underscore.js"></script>
        <script src="../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../" class="icon icon-home"></a> &raquo;</li>
        
      <li>Monitoring Stack with Cephadm</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../_sources/cephadm/monitoring.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../">
          

          
            
            <img src="../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul>
<li class="toctree-l1"><a class="reference internal" href="../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="monitoring-stack-with-cephadm">
<span id="mgr-cephadm-monitoring"></span><h1>Monitoring Stack with Cephadm<a class="headerlink" href="#monitoring-stack-with-cephadm" title="Permalink to this headline">¶</a></h1>
<p>Ceph Dashboard uses <a class="reference external" href="https://prometheus.io/">Prometheus</a>, <a class="reference external" href="https://grafana.com/">Grafana</a>, and related tools to store and visualize detailed
metrics on cluster utilization and performance.  Ceph users have three options:</p>
<ol class="arabic simple">
<li><p>Have cephadm deploy and configure these services.  This is the default
when bootstrapping a new cluster unless the <code class="docutils literal notranslate"><span class="pre">--skip-monitoring-stack</span></code>
option is used.</p></li>
<li><p>Deploy and configure these services manually.  This is recommended for users
with existing prometheus services in their environment (and in cases where
Ceph is running in Kubernetes with Rook).</p></li>
<li><p>Skip the monitoring stack completely.  Some Ceph dashboard graphs will
not be available.</p></li>
</ol>
<p>The monitoring stack consists of <a class="reference external" href="https://prometheus.io/">Prometheus</a>,
Prometheus exporters (<a class="reference internal" href="../../mgr/prometheus/#mgr-prometheus"><span class="std std-ref">Prometheus 模块</span></a>, <a class="reference external" href="https://prometheus.io/docs/guides/node-exporter/">Node exporter</a>), <a class="reference external" href="https://prometheus.io/docs/alerting/alertmanager/">Prometheus Alert
Manager</a> and <a class="reference external" href="https://grafana.com/">Grafana</a>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Prometheus’ security model presumes that untrusted users have access to the
Prometheus HTTP endpoint and logs. Untrusted users have access to all the
(meta)data Prometheus collects that is contained in the database, plus a
variety of operational and debugging information.</p>
<p>However, Prometheus’ HTTP API is limited to read-only operations.
Configurations can <em>not</em> be changed using the API and secrets are not
exposed. Moreover, Prometheus has some built-in measures to mitigate the
impact of denial of service attacks.</p>
<p>Please see <cite>Prometheus’ Security model
&lt;https://prometheus.io/docs/operating/security/&gt;</cite> for more detailed
information.</p>
</div>
<div class="section" id="deploying-monitoring-with-cephadm">
<h2>Deploying monitoring with cephadm<a class="headerlink" href="#deploying-monitoring-with-cephadm" title="Permalink to this headline">¶</a></h2>
<p>By default, bootstrap will deploy a basic monitoring stack.  If you
did not do this (by passing <code class="docutils literal notranslate"><span class="pre">--skip-monitoring-stack</span></code>, or if you
converted an existing cluster to cephadm management, you can set up
monitoring by following the steps below.</p>
<ol class="arabic">
<li><p>Enable the prometheus module in the ceph-mgr daemon.  This exposes the internal Ceph metrics so that prometheus can scrape them.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">mgr</span> <span class="n">module</span> <span class="n">enable</span> <span class="n">prometheus</span>
</pre></div>
</div>
</li>
<li><p>Deploy a node-exporter service on every node of the cluster.  The node-exporter provides host-level metrics like CPU and memory utilization.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">orch</span> <span class="n">apply</span> <span class="n">node</span><span class="o">-</span><span class="n">exporter</span> <span class="s1">&#39;*&#39;</span>
</pre></div>
</div>
</li>
<li><p>Deploy alertmanager:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">orch</span> <span class="n">apply</span> <span class="n">alertmanager</span> <span class="mi">1</span>
</pre></div>
</div>
</li>
<li><p>Deploy prometheus.  A single prometheus instance is sufficient, but
for HA you may want to deploy two.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">orch</span> <span class="n">apply</span> <span class="n">prometheus</span> <span class="mi">1</span>    <span class="c1"># or 2</span>
</pre></div>
</div>
</li>
<li><p>Deploy grafana:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">orch</span> <span class="n">apply</span> <span class="n">grafana</span> <span class="mi">1</span>
</pre></div>
</div>
</li>
</ol>
<p>Cephadm handles the prometheus, grafana, and alertmanager
configurations automatically.</p>
<p>It may take a minute or two for services to be deployed.  Once
completed, you should see something like this from <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">ls</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph orch ls
NAME           RUNNING  REFRESHED  IMAGE NAME                                      IMAGE ID        SPEC
alertmanager       1/1  6s ago     docker.io/prom/alertmanager:latest              0881eb8f169f  present
crash              2/2  6s ago     docker.io/ceph/daemon-base:latest-master-devel  mix           present
grafana            1/1  0s ago     docker.io/pcuzner/ceph-grafana-el8:latest       f77afcf0bcf6   absent
node-exporter      2/2  6s ago     docker.io/prom/node-exporter:latest             e5a616e4b9cf  present
prometheus         1/1  6s ago     docker.io/prom/prometheus:latest                e935122ab143  present
</pre></div>
</div>
<div class="section" id="using-custom-images">
<h3>Using custom images<a class="headerlink" href="#using-custom-images" title="Permalink to this headline">¶</a></h3>
<p>It is possible to install or upgrade monitoring components based on other
images.  To do so, the name of the image to be used needs to be stored in the
configuration first.  The following configuration options are available.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">container_image_prometheus</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">container_image_grafana</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">container_image_alertmanager</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">container_image_node_exporter</span></code></p></li>
</ul>
<p>Custom images can be set with the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">config</span></code> command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">mgr</span> <span class="n">mgr</span><span class="o">/</span><span class="n">cephadm</span><span class="o">/&lt;</span><span class="n">option_name</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">value</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">mgr</span> <span class="n">mgr</span><span class="o">/</span><span class="n">cephadm</span><span class="o">/</span><span class="n">container_image_prometheus</span> <span class="n">prom</span><span class="o">/</span><span class="n">prometheus</span><span class="p">:</span><span class="n">v1</span><span class="mf">.4.1</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>By setting a custom image, the default value will be overridden (but not
overwritten).  The default value changes when updates become available.
By setting a custom image, you will not be able to update the component
you have set the custom image for automatically.  You will need to
manually update the configuration (image name and tag) to be able to
install updates.</p>
<p>If you choose to go with the recommendations instead, you can reset the
custom image you have set before.  After that, the default value will be
used again.  Use <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">config</span> <span class="pre">rm</span></code> to reset the configuration option:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="n">rm</span> <span class="n">mgr</span> <span class="n">mgr</span><span class="o">/</span><span class="n">cephadm</span><span class="o">/&lt;</span><span class="n">option_name</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="n">rm</span> <span class="n">mgr</span> <span class="n">mgr</span><span class="o">/</span><span class="n">cephadm</span><span class="o">/</span><span class="n">container_image_prometheus</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="disabling-monitoring">
<h2>Disabling monitoring<a class="headerlink" href="#disabling-monitoring" title="Permalink to this headline">¶</a></h2>
<p>If you have deployed monitoring and would like to remove it, you can do
so with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">orch</span> <span class="n">rm</span> <span class="n">grafana</span>
<span class="n">ceph</span> <span class="n">orch</span> <span class="n">rm</span> <span class="n">prometheus</span> <span class="o">--</span><span class="n">force</span>   <span class="c1"># this will delete metrics data collected so far</span>
<span class="n">ceph</span> <span class="n">orch</span> <span class="n">rm</span> <span class="n">node</span><span class="o">-</span><span class="n">exporter</span>
<span class="n">ceph</span> <span class="n">orch</span> <span class="n">rm</span> <span class="n">alertmanager</span>
<span class="n">ceph</span> <span class="n">mgr</span> <span class="n">module</span> <span class="n">disable</span> <span class="n">prometheus</span>
</pre></div>
</div>
</div>
<div class="section" id="deploying-monitoring-manually">
<h2>Deploying monitoring manually<a class="headerlink" href="#deploying-monitoring-manually" title="Permalink to this headline">¶</a></h2>
<p>If you have an existing prometheus monitoring infrastructure, or would like
to manage it yourself, you need to configure it to integrate with your Ceph
cluster.</p>
<ul>
<li><p>Enable the prometheus module in the ceph-mgr daemon:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">mgr</span> <span class="n">module</span> <span class="n">enable</span> <span class="n">prometheus</span>
</pre></div>
</div>
<p>By default, ceph-mgr presents prometheus metrics on port 9283 on each host
running a ceph-mgr daemon.  Configure prometheus to scrape these.</p>
</li>
<li><p>To enable the dashboard’s prometheus-based alerting, see <a class="reference internal" href="../../mgr/dashboard/#dashboard-alerting"><span class="std std-ref">启用 Prometheus 报警</span></a>.</p></li>
<li><p>To enable dashboard integration with Grafana, see <a class="reference internal" href="../../mgr/dashboard/#dashboard-grafana"><span class="std std-ref">允许嵌入 Grafana 仪表盘</span></a>.</p></li>
</ul>
</div>
<div class="section" id="enabling-rbd-image-monitoring">
<h2>Enabling RBD-Image monitoring<a class="headerlink" href="#enabling-rbd-image-monitoring" title="Permalink to this headline">¶</a></h2>
<p>Due to performance reasons, monitoring of RBD images is disabled by default. For more information please see
<a class="reference internal" href="../../mgr/prometheus/#prometheus-rbd-io-statistics"><span class="std std-ref">RBD IO 统计</span></a>. If disabled, the overview and details dashboards will stay empty in Grafana
and the metrics will not be visible in Prometheus.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>